1. Field of the Invention
This invention relates to networked systems, and specifically to an improved system, method, and computer-readable instructions for scheduling, transporting, and receiving inbound packets efficiently in networks with cyclic packet scheduling.
2. Discussion of the Background
Cyclic real-time networks—In a plant or machine where there are several actuator and/or sensor devices, there are many advantages to be gained from connecting these to a central controller using a serial network. These advantages include modularity, reduction in wiring, more extensive central monitoring and improved serviceability. To permit the automation of high performance machines it is generally necessary to issue commands to the actuators frequently and, similarly, to gather data from the sensors frequently. Furthermore it is advantageous for the commands to take effect simultaneously at all of the actuators and similarly for the instant at which measurements take place to be simultaneous at all of the sensors; this is particularly true of motion control systems. Specific serial networks have been devised (e.g., IEC61491-2002 “Adjustable Speed Electric Drive Systems Incorporating Semiconductor Power Converters” (SERCOS™); Interests Group SERCOS interface e. V.: “Specification SERCOS III interface Version 1.20” {Dec. 22, 2004} {from www.sercos.de} U.S. Pat. No. 7,024,257, U.S. Pat. No. 7,143,301, SynqNet) or adapted from standard data transfer networks (e.g., IEC PAS 62408 First edition 2005-06 “Real-time Ethernet Powerlink (EPL)”) in order to meet these requirements; the essential characteristics of these networks are that: (a) a controller sends at least one packet containing command and/or other data to each slave sensor/actuator node according to a cyclic, repeating timetable (sometimes called isochronous communication in the literature), (b) the measurements and/or other data are sent from each slave sensor/actuator node to a controller according to a cyclic, repeating timetable, (c) the controller has a master timer and each slave sensor/actuator node on the network has a means of re-constructing the controller's timer, for example by adjusting the local timer on receipt of a timing packet, (d) a high-speed physical layer is used (such as [e.g., ANSI/IEEE Std 802.3u-1995: “IEEE Standards for Local and Metropolitan Area Networks: Supplement to Carrier Sense Multiple Access with Collision Detection (CSMA/CD) Access Method and Physical Layer Specifications Media Access Control (MAC) Parameters, Physical Layer, Medium Attachment Units, and Repeater for 100 Mb/s Operation, Type 100BASE-T (Clauses 21-30)”], dictating that point-to-point connections rather are used (rather than a physically bussed network) which in turn implies the use of repeaters either in the nodes themselves or using hubs.
Several networks are organized in this way (SERCOS I & II, SynqNet, Ethernet-Powerlink, SERCOS III). Such networks can be described as “cyclic real-time networks”.
Operation of cyclic real-time networks—In order to aid the readability of this document it is helpful to define some terms. ‘Node’ means slave node on the network (other than the controller) that optionally has sensor/actuator functionality. ‘Inbound’ means from any node to the controller. ‘Outbound’ means from the controller to any node. ‘Port’ means the circuitry permitting a connection to the network (connector, any isolation means, protection components, transceiver). ‘Upstream port’ means the network connection port of a node through which inbound packets are sent to the controller. ‘Downstream port’ means the network connection port or ports of a node through which outbound packets are sent to a further node or nodes. ‘Switch’ means layer 2 network switch. ‘Jitter’ refers to the unpredictable and fluctuating delay of a packet. ‘Enumeration’ means the discovery of nodes and the configuration of nodes by the controller via the network. ‘Optimum Inbound Throughput’ or simply ‘OIT’ is the name of this invention. ‘PHY’ means a network transceiver. Descriptions of the network physical layer will use 100BASE-T terminology.
Cyclic real-time networks can be full-duplex or half-duplex; FIG. 8 shows an example full-duplex cyclic, repeating timetable (whereas FIG. 19 of IEC PAS 62408 First edition 2005-06 “Real-time Ethernet Powerlink (EPL) shows an example half-duplex cyclic, repeating timetable).
A system comprises a controller, carrying out control/monitoring of the systems and one or more nodes that have network functionality and may also have actuation/measurement functions. The nodes are connected to the controller via a high-speed network.
In FIG. 1 the controller 0105 contains a processor sub-system 0101 which executes the control program for loop closure and the sequencing of actions, the controller network logic 0102 which organizes the sending and receiving of packets from the network and houses the master timer for the system (the controller network logic 0102 is typically an FPGA or an ASIC), and at least one 0103 or more 0104 network port(s) comprising a transmitter-receiver circuit such as a 100BASE-T PHY device with associated components.
Referring to FIG. 2, each node 0205 contains the node network logic and buffer circuit 0202 which sends and receives packets from the network as well as housing the node's re-construction of the controller's timer (it is typically an FPGA or an ASIC), at least one but more usually two or more 0201, 0203 network port(s) comprising a transmitter-receiver circuit such as a 100BASE-T PHY device with associated components. Optionally each node can also include a sensor/actuator circuit 0204 which carries out the measurement and actuation function of the node.
The controller and nodes can be interconnected on a network using a line connection topology (see FIG. 6).
In all topologies the packets reach farther nodes by passing through repeaters that are located within intermediate nodes or within dedicated repeating devices. At its simplest (see FIG. 3) a repeater comprises a PHY device 0201 for one port, a PHY device 0203 for the second port, an elasticity buffer 0302 connecting the ports in the upstream direction and a second elasticity buffer 0303 connecting the ports in the downstream direction. The elasticity buffers do not buffer the IDLE symbols that occur between packets, this ensures that the entire packet is forwarded, the number inter-packet IDLE symbols is adjusted to fill the inter-packet gap (i.e. IDLE symbols are appended to the transmitted packet until a further packet is received), in this fashion a packet can be received on one port at one clock rate but can be transmitted from the other port at a slightly different clock rate.
The repeater can be incorporated into a node having sensor/actuator functionality (see FIG. 4) by adding a multiplexer 0404, the remainder of the node network logic 0405 and the sensor/actuator circuit 0204. At all times the node network logic 0405 can receive packets 0406 from the upstream PHY 0201 and these packets are also forwarded to the downstream PHY 0203. When the signal XMIT_A (0401) is TRUE the multiplexer steers packets emitted by the node network logic 0405 along signal path 0402 to the upstream PHY 0201. When the signal XMIT_A is FALSE the circuit acts as a repeater in both directions.
Referring to FIG. 6 and FIG. 8, when network communication is cyclic, the controller 0105 sends outbound packets 0801, 0802, 0803, 0804 to the nodes 0602, 0603, 0604 according to a cyclic timetable. One 0801 of the outbound packets is used by the nodes to re-construct the controller's timer at each node, in a practical system, the timer re-construction circuit can optionally be enhanced by making use of information such as the delay time of the packet in reaching the node in question and the nominal communication cycle period 0809. The nodes send inbound packets 0805, 0806, 0807 back to the controller in accordance with a previously configured cyclic timetable using the re-construction of the controller time.
Scheduling of packets in cyclic real-time networks—In cyclic real-time networks it is essential to ensure that the packets can not collide as the consequential loss of data would impair the ability of the controller to control/monitor the system.
Outbound scheduling—Outbound packets are separated by inter-packet gaps 0813, for which the minimum possible value is determined by various factors: one of these is that if the nodes have successively slower clock oscillators the packets will bunch more closely together as they pass through each node, another is the need to insert some idle time on each network segment so that receiving PHYs can lock their descrambler circuits. In a practical system the minimum outbound inter-packet gap used by the controller can be as short as eight IDLE symbol times.
In the outbound direction all packets are sent by the controller and, provided sufficient idle time is inserted between the packets to accommodate the variation in local oscillator frequencies, collisions can not occur.
There is some flexibility in the outbound schedule of packets. The timing packet 0801 must have a fixed position within the communication cycle 0809 to ensure the correct operation of the timer reconstruction circuit. The total set of outbound packets (0801, 0802, 0803, 0804) must fit within the communication cycle 0809 but, subject to these conditions, the command packets (0802, 0803, 0804) can vary in length or sequence or destination.
Inbound scheduling—In the inbound direction the nodes must send their packets in non-overlapping time-slots; in order to achieve this, the controller has a timer and each node locally reconstructs the controller's timer and each respective node sends its inbound packet(s) in their allotted time slot(s) within the cyclic timetable. Inbound packets are separated by inter-packet gaps 0808. The minimum duration of these gaps is constrained not only by the need to accommodate slight differences in the nodes' clock oscillators and to supply the PHYs with IDLE symbols but more significantly to cope with the uncertainties in the node's re-construction of the controller timer.
There is less flexibility in the inbound schedule of packets; each of the set of inbound packets (0805, 0806, 0807) must fit within the allocated slot times (0810, 0811, 0812). Subject to this set of constraints, each inbound packet can be shorter than the slot time it occupies and similarly the slot times 0810, 0811, 0812 could be used for different nodes from cycle to cycle (e.g. nodes 0, 1, 2 on one cycle and nodes 3, 4, 5 on the alternate cycle).
Timing uncertainty in cyclic real-time networks—Nodes can be connected to the controller using networks supporting various topologies such as a tree FIG. 15 or a line FIG. 6 or a ring FIG. 7. As the node count rises, the delay endured by packets to reach the farthest nodes become significant, especially for the line and ring topologies. The reconstruction of the controller's timer can be made more accurate by taking account of the transmission delay time of the packet to the destination node, this delay comprises the propagation time of the network cabling and the delays of the PHYs and repeaters in each intervening node within the chain. However various timing uncertainties (limitations in the accuracy of the reconstructed clock, variations in local oscillator frequencies and uncertainties in the measurement of the transmission delay time) dictate that the time slot should be longer than the duration of the inbound packet by a contingency time.
These timer uncertainties accumulate with each network cable and repeater traversed by the timing packet; the uncertainty of the re-construction of controller timer at node 2 (0604) is three times that at node 0 (0602). Therefore the contingency time should be longer in networks where many nodes are connected in series and in practice this will seriously reduce the data efficiency of the network in the inbound direction and will therefore limit the network cycle rate that can be achieved. The problem is exacerbated by the fact the cyclic packet loading is generally greater for the inbound direction than for the outbound direction; in most systems there is more data to monitor than to control. In a practical system the minimum inbound inter-packet gap 0808 can be as long as one symbol time per cascaded node plus the minimum of eight IDLE symbol times: e.g. 258 symbol times (10.32 μs when using the 100BASE-T physical layer) for 250 nodes connected in series
Cyclic real-time networks then, suffer progressive reductions in the accuracy of each node's reconstruction of the controller timer as the number of intervening repeaters increases and consequentially the utilization of the network in the inbound direction deteriorates as the ability to accurately schedule inbound packets is progressively impaired.
Cyclic real-time networks also have inflexible inbound packet schedules.
Let us now turn our attention to proposed remedies to improve the inbound utilization of cyclic real-time networks and bring flexibility to the inbound packet schedule.
Hubs—One approach to improve the inbound utilization of cyclic real-time networks is to limit the number of intervening repeaters by using repeating hubs to create tree topologies. Powerlink, for example, (IEC PAS 62408 First edition 2005-06 “Real-time Ethernet Powerlink (EPL) page 292 et seq.) uses a combination of repeating hubs and internal repeaters within the sensor/actuator nodes themselves to limit the number of intervening nodes required to reach the farthest node. The structure of a four-port repeating hub is shown in FIG. 14 and operates as follows: the central logic 1407 simply forwards symbols received on any port to all of the other ports and with minimal delay. For example using a tree structure with two tiers of 17-port hubs (one on the first tier and seventeen on the second tier—requiring eighteen hubs in total) it is possible to fan out to more than 250 nodes with only two repeaters in the path to the most distant node from the controller. There are however some significant disadvantages to this solution; firstly hubs are inherently half-duplex whereas there are both throughput and latency advantages to be gained from using full-duplex communication, secondly hubs have been wholly superseded in the mass-market by switches and are thus difficult to obtain and therefore expensive, thirdly hubs are restricted to tree topologies and these inherently can not support the re-routing of packets in the event of a fault that is possible in a ring, fourthly the physical arrangement of the plant or machine may make a tree topology physically inconvenient by obliging multiple cables to run in trunking or cable retractors where only one or two cables would be required for line or ring topologies (this same objection also applies to the use of free-standing switches). The use of repeating hubs is therefore not an ideal solution.
Standard Switches—Another category of partial remedies makes use of standard Ethernet switches. These devices also have a structure of the form shown in FIG. 14 but the operation of the central block 1407 now has at least four, complex functions: selective bridging, store-and-forward, cut-through and flow control.
Selective bridging means that packets are forwarded only to the port to which the destination node is attached. The switch can be configured to steer particular packets to particular ports (in the case of managed switches) or the destination port can be inferred by the switch itself during a learning process that requires the inspection of incoming source addresses (in the case of unmanaged switches). Note that a learning process compromises determinism.
Store-and-forward means that the incoming packet is held in a buffer for re-transmission when the destination port has been determined and when the destination port has become free.
Cut-through attempts to speed-up the store-and-forward process by inspecting the destination address of a packet as it arrives rather than waiting for the whole of the packet to be received.
Flow control causes a pause frame to be sent to a connected node in the event that the switch is too busy to handle further packets on that port.
Switches do promote efficient utilization of the network; they are full-duplex and they do remove the need for the contingency times in the inbound packet schedule because no packet collision will occur even if more than one node is transmitting its inbound packet simultaneously. Switches are complex devices and typically require a processor in combination with additional logic to accelerate packet handling but, owing to mass-manufacture, they have become inexpensive. Despite their many merits in the realm of data networks, switches are not suitable for use in cyclic real-time networks for two main reasons. Firstly, the packet must be a standard Ethernet packet (and thus carry the burden of the 6-byte source address, 6-byte destination address, 2-byte Ethertype field and a minimum of 46 byte data payload) in contrast to the compact packets used on high-performance cyclic real-time networks and in consequence the cyclic rate of the network (for a given number of nodes) is relatively low and latencies are also mediocre in both the outbound and inbound directions. Secondly, the outbound packets suffer an unpredictable and fluctuating delay in reaching the nodes that is in the order of tens of micro-seconds, therefore the packet used for correcting the reconstructed timer in each nodes suffers significant jitter and therefore the reconstructed timer may simply be too inaccurate for measurement and control functions (where timing accuracy on the order of 1 μs may be required). Thus ordinary Ethernet switches can not be used to create cyclic real-time networks that are state-of-the-art in respect of data-efficiency, cyclic rate, latency or timing accuracy.
Modified Switches—Two networks set out to solve the problem of poor timing accuracy by using modified Ethernet switches.
Ethernet/IP (EtherNet/IP™ Specification Volume Two: “EtherNet/IP Adaptation of CIP” (from www.odva.org)) uses IEEE 1588 (IEEE-1588: 2003, “Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems”) to correct the timer in the node. This solution mandates the use of IEEE 1588 circuitry in the controller, the nodes and any intervening free-standing switches.
Profinet/IRT (“Automating with PROFINET: Industrial communication based on Industrial Ethernet: Industrial Communication Based on Industrial Ethernet”, by R. Pigan and M. Metter, published by Wiley, ISBN-13: 978-3895782565) has a special timer correction scheme. Simply put, Profinet/IRT uses special switches based on an ASIC to prioritize, on a cyclic basis, those packets that are participating in cyclic, real-time communication. This solution obligates the use of Profinet/IRT circuitry in the controller, the nodes and any intervening free-standing switches.
Both Ethernet/IP and Profinet/IRT do solve the problem of having contingency times in the inbound packet schedule but only by adding circuitry of considerable complexity and expense. However neither network attempts to alleviate the other performance limitations that arise from using standard Ethernet packets (poor data-efficiency, low cyclic rates and longer latencies).
A number of patents exist that describe store-and-forward, cut-through, and other scheduling and transporting techniques, such as U.S. Pat. Nos. 4,933,933, 5,307,345, 5,633,876, 6,094,434, 6,144,668, 6,356,558, all incorporated herein by reference.